feat(18): drop the local utils.py - #26
Merged
Merged
Conversation
The 35 modules that imported the local utils now take it from dm_job_utilities, which provides all of log, expand_path, read_delimiter, is_type and calc_geometric_mean. Importing the module rather than the names (`from dm_job_utilities import utils`) leaves every utils.x call site unchanged. This repo is the only consumer that actually reached the two functions #18 flags as blockers: - round_to_significant_number (5 uses, all in dmpk/pk_tmax_cmax_sim.py) is replaced by sigfig.round, as job-utilities' README directs. The local implementation leaned on builtin round() and so inherited its half-way behaviour: 2.675 rounded to 2.67 and 1.005 to 1.0, where sigfig gives 2.68 and 1.01. Over 12000 randomly generated PK-plausible values, none differ - simulated values do not land on exact decimal boundaries - so this is a correctness fix with no expected change to real output. Wrapped locally to pass warn=False, since sigfig warns when a value carries fewer significant figures than requested and the previous implementation was silent. - get_path_from_digest (used by assemble_conformers.py and prepare_enum_conf_lists.py) was deliberately dropped from job-utilities as obsolete, but the sharded directory layout it describes is still what one script writes and the other reads. It moves to digest_utils.py rather than being inlined into both. Every Dockerfile installing job-utilities is bumped to >= 1.4.0: dm_job_utilities.utils first appears in 1.3.0, so the images pinned at 1.0.1 and 1.1.1 would have failed at import. sigfig is added to Dockerfile-dmpk, which did not carry it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three separate faults, all mine, all in the Dockerfiles: - Five images COPY utils.py explicitly (fns, dmpk, plants, jaqpot, mordred); with the file gone the COPY could not resolve. Removed. The images that glob (COPY *.py) were unaffected, and that glob is also how digest_utils.py reaches vs-prep, which is where both its consumers run. - Changing the pins from == to >= left them unquoted, so /bin/sh read the > as a redirection: `pip install foo>=1.4.0` installed foo unpinned and wrote stdout to a file named "=1.4.0". Every spec containing >= or <= is now quoted. The builds that passed had been silently installing unpinned. - Dockerfile-rdock never needed bumping: prepare_rdock.py imports no utils at all, and the image is Python 2 (rdock:2013.1), where a modern job-utilities cannot install. Reverted to its original pin. Also bumps the remaining im-rdkit-utilities==1.0.0 pins to >= 1.1.2. That version pins job-utilities to ==1.3.0, so it conflicts with >= 1.4.0 - the same conflict rdkit-utilities 1.1.2 was released to fix. It only surfaced once the quoting was corrected. All nine images build locally, resolve dm_job_utilities.utils (and rdkit_utils where used) at the intended versions, and carry no stray redirect file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second and final PR for this repo under InformaticsMatters/squonk2-jobs#18,
after #24 and #25 which adopted the shared CLI helpers.
Summary
The 35 modules importing the local
utilsnow take it fromdm_job_utilities, which provides all oflog,expand_path,read_delimiter,is_typeandcalc_geometric_mean. Importing the modulerather than the names —
from dm_job_utilities import utils— leaves everyutils.xcall site unchanged, so the diff is one line per file.utils.py(157 lines) is deleted. Net −206/+109.The two #18 blockers
This repo is the only consumer that actually reached them.
round_to_significant_number→sigfig.round5 uses, all in
dmpk/pk_tmax_cmax_sim.py(Tmax, Cmax, Kel, Ka, V/F at 3 s.f.).The local implementation leaned on builtin
round()and inherited its half-waybehaviour — job-utilities' README calls it inaccurate, and that checks out:
sigfig2.6752.672.681.0051.01.01Over 12,000 randomly generated PK-plausible values, none differ. The
divergence needs a value sitting exactly on a decimal boundary, which typed
constants produce and simulation output effectively never does. So this is a
correctness fix with no expected change to real output — but it is a numeric
change to a pharmacokinetics Job, which is why it was kept out of #24.
Wrapped locally as
sigfig_round()to passwarn=False:sigfigwarns when avalue carries fewer significant figures than requested (e.g.
7.0at 3 s.f.),which is common here and was silent before. Without it the Job log would fill
with
UserWarning.get_path_from_digest→digest_utils.pyUsed by two scripts,
assemble_conformers.pyandprepare_enum_conf_lists.py. job-utilities dropped it as obsolete, but thesharded directory layout it describes is still what one script writes and the
other reads back — it is not obsolete here. Moved to its own small module rather
than inlined into both. Verified identical to the original across 2,020 cases
(500 real digests × 4 chars/levels configurations, plus degenerate inputs).
Dockerfiles
Every Dockerfile installing job-utilities is bumped to
>= 1.4.0.This was load-bearing, not tidying:
dm_job_utilities.utilsdoes not existbefore 1.3.0 — I checked the published wheels.
Dockerfile-dmpkandDockerfile-rdockwere pinned at1.0.1, andDockerfile-fnsandDockerfile-plantsat1.1.1, so all four would have failed at import.sigfigis added toDockerfile-dmpk, which did not carry it (Dockerfile-prepalready did, for
sa_score.py).Verification
Every
utils.*reference in the repo was resolved against the shared moduleprogrammatically — no unresolved names.
Output parity against
main, same inputs:rdkit_props.pyrdkit_dedup.pysa_score.pycluster_butina.pyle_conformers.pymax_min_picker.pydiffers — but it differs against itself run twice onmain. It's non-deterministic, so unrelated to this change.assemble_conformers.pyandprepare_enum_conf_lists.py(the digest consumers)import cleanly.
dmpk/pk_tmax_cmax_sim.pycan't be imported locally(
matplotlibabsent — identical onmain), so its changed rounding function wasexercised directly.
All 36 touched Python files parse.
jotewas not run — needs the affected images rebuilt. Worth a CI run,particularly for the dmpk and rdock images whose pins moved furthest.
🤖 Generated with Claude Code